We propose a model to learn visually grounded word embeddings (vis-w2v) tocapture visual notions of semantic relatedness. While word embeddings trainedusing text have been extremely successful, they cannot uncover notions ofsemantic relatedness implicit in our visual world. For instance, although"eats" and "stares at" seem unrelated in text, they share semantics visually.When people are eating something, they also tend to stare at the food.Grounding diverse relations like "eats" and "stares at" into vision remainschallenging, despite recent progress in vision. We note that the visualgrounding of words depends on semantics, and not the literal pixels. We thususe abstract scenes created from clipart to provide the visual grounding. Wefind that the embeddings we learn capture fine-grained, visually groundednotions of semantic relatedness. We show improvements over text-only wordembeddings (word2vec) on three tasks: common-sense assertion classification,visual paraphrasing and text-based image retrieval. Our code and datasets areavailable online.
展开▼
